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Abstract 

Across the various programs of study, portfolios serve many functions and purposes. Electronic 
portfolios have been used as a culminating product for students in our Master of Library Science and Master of 
Education in Instructional Technology at East Carolina University. Two conclusions were drawn from this 
study - (1 ) authentic assessment using portfolios is useful for facilitating reflective thinking that results in self- 
regulated learning, and (2) student products in the form of electronic files can be archived, indexed and used as 
evidences in program evaluation. 



Introduction 

The need for authentic methods for assessing educational outcomes has led to a move from 
quantitative measures to a more an open-ended qualitative format. Through the use of portfolios, students are 
able to select and evaluate their own products of learning and present these for final certification before entering 
their respective professions. The portfolio provides, not only a method for assessment, but also is a catalyst for 
learning. These measures are dependent on process, as well as outcomes from the learning ejperience. Using 
the processes for selection, evaluation, and alignment of work samples with a particular standard for excellence 
would likely help the student transfer what is learned within the laboratory setting to the real world of work 
(Wolf, 1998). 



Portfolio Formats and Functions 

One of the earliest reported uses for the portfolio was in the visual arts for the purposeful collection of 
one’s best work (Friedman Ben David, et ah, 2001). Many schools continue the use of the showcase portfolio 
to display exemplary student products as a culminating experience and for prospective employers (Baltimore & 
Hickson, 1996). When using this type of showcase portfolio, there are no comparisons between entry-level 
work and expert performance by the student. Thus, the contents do not provide substantial evidence that training 
or education has had notable impact on the student. 

Because of our society’s evolution from industrial- lockstep work environments to more open-ended- 
informational environments, there is an important emphasis on authenticity in student learning experiences. 
Employers are looking for candidates who can examine their environment, draw logical conclusions, and 
develop problem-solving strategies based on a given situation (Weiner, 2000). This is accomplished through the 
use of situated problem solving and authentic assessment of outcomes from learning (Young, 1995). 

Another important use of portfolio is program evaluation. Portfolio assessment requires the careful 
analysis of program goals and objectives and how these are transferred to the course activities and assignments. 
Student artifacts should mirror program goals. When this is evident within the portfolio, evaluation of the 
program of study is facilitated (Koretz, 1992; Payne, 2000). 

Portfolios and Learning 

Regardless of format, function, or purpose, portfolios can be classified as either capstone experience 
(showcase) or a record of process in learning (assessment portfolio). The capstone portfolio includes stand- 
alone evidence for mastery of program objectives, examples of student’s best work, and documents from 
culminating experiences. Typically, accomplished students who are about to enter their chosen profession are 
associated with the "capstone" category. Programs that require the capstone (or showcase) portfolio should 
specify work samples that will be of interest to prospective employers and artifacts that are cognizant of the 
profession. In addition, expectations and standards for best practices must be clearly communicated to the 
student (Skawinski & Thibodeau 2002). 

A second category is the "process or learning portfolio.” The contents represent processes for cognitive growth, 
interrogation of the learning environment, self-assessment using recognized standards, and transference of 
learning to the workplace. For the instructor or faculty member, there is a responsibility to the student to 
monitor cognitive growth as a result of assigned projects and field experiences. By providing cognitive 
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scaffolds for reflection, self-assessment, and strategies for making changes, the process approach to learning is 
exemplified within the contents of the portfolio (Murphy, 1997). The instructor/assessor acts as a guide or 
proctor during the development of the portfolio, and models collaborative practices as mentor and mentee work 
together to select artifacts and other evidence that show growth over time. An important role of the 
instructor/assessor is to provide critical commentary and invite the student to defend, justify, and make 
adaptations to his or her work samples. The dialog between faculty and student can be very productive during 
these advising sessions. 

In addition to evaluation of student learning, either process or capstone portfolios, are useful for the 
analysis of a program of studies. Program evaluation is facilitated when key players for the portfolio process are 
committed to the necessary time requirements, practices for self-evaluation, and adoption of authentic 
assessment methodologies (Johnson, et ah, 2000; Campbell, et al., 2000; Baume & Yorke, 2002). For the 
student, this means gaining skills as a reflective practitioner. He or she must be willing to adopt the process- 
approach for learning. This means entering the program as a novice, accepting critical commentary, working 
through revisions, and planning for the future. For the assessor, it requires a commitment for adequate time with 
students for mentoring and modeling for reflective practices (Freidus, 1996). In addition, there must be time 
devoted to careful planning for program goals, objectives, and classroom activities that reflect these objectives. 

Reliability and Validity Using Assessment Portfolios 

A major consideration with implementation of portfolio assessment is reliability of measures and 
validity of the assessment. Latrobe and Lester (2000) discovered in their Library Science program that 
establishing valid measures is difficult because competent performance may... “vary in depth, in approach, and 

in the specificity of the professional work addressed ”, Although it is difficult to gather data related to 

reliability in portfolio assessment, (Friedman Ben-Davis & et al., 2001) as a result of this review, several studies 
were identified, and were supportive of, portfolios for assessment purposes (Baume & Yorke, 2002). Other 
reports are not as encouraging (Koretz, 1998). There are, however, certain characteristics that were apparent in 
programs with reports for reliable use of portfolio assessment. Measures are reliable when there is evidence that 
portfolio contents represent an accurate picture of the program goals/objectives or other recognized standards 
for the profession (Bullock & Hawk, 2001; Campbell, et al., 2000; Pitts, Coles, & Thomas, 2001; Routledge & 
Willson, 1997). In addition, correlations among assessors’ scores are high when there is evidence for clear-cut 
indicators of acceptable performance. Another characteristic associated with the reliable use of portfolios is the 
selection of artifacts; either specified in advanced or self-selected by the student, these should be representative 
of program goals and objectives. Along with specific criteria, there are standardized levels of difficulty and 
consistency in characteristics of the evidence or artifacts. Reports from the literature suggest that correlations 
can be very low when there are inconsistencies among artifacts. Reliability measures were high when clear-cut 
criteria for evaluation had been agreed upon by assessors and performance indicators were representative of the 
standards or competencies adopted by the program. Reliability measures were also high with reports for 
sufficient training of assessors. 

One strategy used by programs to ensure strong reliability and validity measures was through 
collaborative meetings to reach consensus on scores. When planning implementation of portfolios, faculty 
should meet to analyze the strength of relationships between program goals, performance indicators, and quality 
of the portfolio contents. Typically, there are three assessors assigned to a team. Contents are evaluated by the 
first 2 assessors who score independently. When there are wide differences in scoring, a third assessor reads and 
evaluates only those sections with disparate scores. ( Baume & Yorke, 2002; Friedman Ben David, et al, 2001; 
Davis, et al., 2001; Skawinski & Thibodeau, 2002) Of the studies reviewed, the third reader usually scores in 
agreement with “pass” or “marginally pass”. Careful alignment of program objectives with course activities, 
clear communication of expectations aligned with these objectives, and a specified standard for formatting and 
presentation of the portfolio were all associated with valid and reliable measures. 

Purpose of the Study 

The purpose of this study was to identify specific advantages for the use of assessment portfolios and 
how these advantages might support the graduate programs' goals and objectives. To satisfy these purposes, a 
formative evaluation of student reflections and faculty satisfaction for use of portfolio as a method for 
assessment was conducted. Three main areas were the focus of the study- student response to portfolios with 
reflective writing as a metacognitive process, validity of the evaluation methods, and solutions to management 
of data for program evaluation. 
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Self-regulated Learning and Reflective Writing 

As a method for evaluating student response to electronic portfolios, end of course surveys were sent 
to students enrolled in the graduate level course "Development of Electronic Portfolios”. The course was 
offered as an elective for students enrolled in any of the department's three graduate programs. The two main 
objectives for the course were (1) guide students through the reflective writing process, and (2) teach skills for 
file management and design of the portfolio using web editors. Students also increased their comfort level in the 
use of tools for electronic file transfer and online development of the course projects. The course received high 
evaluations and has been requested for subsequent semesters. There are two reasons for this . First, students 
recognized the need for developing their portfolios with skill, accuracy, and as a true measure of academic 
achievement. The course seemed to meet this need. Secondly, students were able to interact with the instructor 
(also their faculty advisors), and with each other, for feedback on their progress in developing the portfolio. 
Dialog was constructive within a nonthreatening environment thus students experienced a formative assessment 
of their work contributing to greater confidence for completing the portfolio. 

An important concept built within the course was self-regulated learning using the reflective writings 
for each artifact. Self-regulation as a method for achieving learning goals leads to increased motivation, self- 
monitoring, attention control, application of learning strategies, and other metacognitive thinking processes 
(Ormrod, 1999). Using a theoretical base for self -regulated learning (Zimmerman & Bandura, 1994), questions 
for the survey were designed to gather information in how students used the development of their portfolios and 
reflective writings for the following thinking processes - focus their thinking and goal-setting, self-assessment 
of quality of work based on standards, and time management strategies. Table 1 provides a summary of 
comments related to each main area of self-regulated learning. 

Table 1. Student comments related to self-regulated learning and use of portfolios. 

For focused thinking, students commented: 

Writing the reflection is usually a pretty long process for me that takes a few rough 
drafts before the final draft is complete. It is almost like I am constantly reflecting 
on my reflection, if that makes any sense. Hopefully I will become better and more 
focused on writing my reflections with more experience of doing so. ... The 
portfolio definitely forces one to focus upon what one has accomplished. The 
reflective writings, in particular, has helped me to focus on what I did, why it was 
important, how it could be improved, and how my future will be impacted by what I 
did. 

For self-assessment of progress, students commented: 

I do think that my portfolio will help me assess my progress as I complete more 
courses during my degree. Hopefully the quality of my artifacts and reflections will 
improve each course and by adding those to my portfolio will help me see if this 
happens or not. I also think that my portfolio layout will probably change over the 
course of completing my degree. I will reflect as well on the layout and design to 
hopefully make it look even better and more professional for others get a clearer 
picture of who I am and what I have to offer. . . . The reflective writing process 
enabled me to assess my progress (where I’ve been) and decide the future path to 
take (where I want to go).. . . Through reflection I have learned to assess the 
strengths and weaknesses of a project, which in turn helps me decide how I can plan 
the next project more efficiently. 

For Time -management, students commented: 

I do feel that the portfolio has contributed to my planning and organization, but do 
not feel the reflections have done the same. . . I would have completed the portfolio 
in a timely manner regardless of disciplined planning. I have difficulty with time 
management, and I cannot hones tly say that the portfolio has contributed to better 
organization. . . . the reflective writing did not affect time management. 

Students reported less than favorable improvement in time management associated with 
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reflective writing and the portfolio process. It likely that additional time needed to learn new skills for 
developing the electronic portfolio had a negative effect on their perceptions for time management. In 
addition, students may have misinterpreted the survey question and answered as time required to 
develop the portfolio rather than the actual intent - improved time management skills for course 
assignments as a result of the reflective writings. 

Reflective Writings for Self-assessment 

Each artifact in a student's portfolio must be aligned with a standard, professional competency, or 
program objective. Program areas vary in their decision to allow students' self-selection of artifacts. Reports in 
the literature suggest student self-selection is beneficial for self-assessment and monitoring of one's own 
learning. However, for program evaluation, our program area faculty have consistently recommended pre- 
selected artifacts that can be archived and used for program evaluation and accreditation purposes. Each core 
course within their degree program includes an assignment that is designated for the student's final portfolio. 

The student must analyze the requirements and final outcomes from the assignment to determine appropriate 
standards, competencies, or objectives. Students are taught to describe the what, how, and where of an 
assignment, describe outcomes ("are you satisfied, what would you change about your final product?") and to 
align their assigned work sample as evidence for meeting the standard, objective, or competency. Students must 
also reflect on how the assignment will impact their future career goals. The rationale for including this 
requirement to the reflection is to aid in transfer of skills and concepts to future work environments. In Table 2. 
is a student's reflective writing that provides clear and consistent evidence that a competency has been met and 
that transfer to the workplace is highly probable. 

Table 2. Sample of Student Reflective Writing. 

LIBS 6014 Introduction to Reference. Reflection MLS Program Objective 3:To answer 
reference questions using print and electronic resources 

In LIBS 6014, students were asked to “compile a pathfinder of reference sources on a specific 
topic designed for a specific user group and based on a review of the existing literature in any 
given subject field.’' I chose to create a pathfinder on plants since I am teach third grade 
Science, and this would be useful to me in my classroom since the North Carolina Standard 
Course of Study includes plant adaptation and growth as Goal 1 in third grade. . . . Since 
creating this pathfinder, I have been compiling pathfinders on other thematic units that I 
currently teach as well as others that I think will help my colleagues. I have shared my 
pathfinders with my colleagues and media specialist at my school. Whenever a teacher begins 
a new thematic unit, they now come to me for guidance. 

Artifact: Pathfinder for 6018: 

"Election Year Politics: The Political Right vs. the Political Left: What Does it All Mean?" 

This project exemplifies MLS Objective 3: To answer reference questions using print and electronic 
resources 

A pathfinder is a useful tool that points people to sources of information. I chose this topic 
because it is very relevant to American culture in this election year. This pathfinder will help 
someone without a political background read and comprehend an article about the election. 

I found most of my sources in Joyner Library on the ECU campus, but many, if not all, of 
these sources can be found at any public or academic library. . . In the future I might change 
the pathfinder by adding a section aimed at school-age children or by re-writing it with that 
audience in mind. . . .If I added this section to the pathfinder or re-wrote the whole thing, I 
would actually strengthen my mastery of Objective 3 because I would be helping answer even 
more reference questions using print and electronic sources. . . This project was a valuable 
one for me because I will be creating pathfinders often as a school media specialist. Overall, 
the project was very time consuming, but in the end, I was able to acquire or hone the above- 
mentioned skills for future use in my professional field. 
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There is one disadvantage for designating a specific project for the portfolio. Students may devote an 
unbalanced amount of time and energy to the identified project and neglect other equally important 
assignments/projects. See this student's comment and how she has spent the majority of her time on designated 
projects - 

I initially thought that we would undergo several projects in each class and we would choose 
our best to be presented in our portfolio. However after these last two semesters, I see there’s 
only one main project per course so now I contribute more planning time to that one project 
b/c I know it will be presented in the portfolio and it needs to be my best work. 

This student unwittingly admitted extra time for the portfolio project as she responded to the survey question on 
"time management". 



Inter- rater Reliability 

A major challenge was bringing faculty to consensus on collection methods and format for data used as 
evidence in the portfolio. Initially, faculty envisioned the evaluation process as a duplication of efforts. "Why 
should I evaluate student products if they've already been evaluated and graded?" was the general response. 

With time, faculty began to see the evaluation of the reflective writings as one valid measure of students' 
mastery of key concepts and skills in the courses. 

Simple logistics such as how and when to establish evaluation committees also presented obstacles. 
Rubrics had to be developed with clear indicators that were directly aligned with either the Program Objectives 
or standards and competencies recommended by state agencies and learned societies. Much deliberation was 
required before all faculty were in agreement. Once this was accomplished, committees were formed and 
student portfolios were efficiently and reliably evaluated at the end of each semester. Students were notified of 
deadlines well in advance of due dates. This allowed faculty adequate time for the evaluation process and 
provided the student time to make necessary revisions or updates to their final product. 

Each portfolio was evaluated by at least three faculty members. Each indicator within the rubric 
included a scale of 0 to 5. Total points possible for each indicator being 5 and total points for the entire portfolio 
being equal to the number of indicators for the entire rubric. The expectation was that 100% of the students 
reached mastery following recommendations for revisions. 

Recommendations 

Two advanced degree programs at East Carolina University have used electronic portfolios as the 
culminating project before graduation. The Master of Library Science began using the electronic portfolio in 
2001 (Brown & Boltz, 2002), and more recently, the Master of Education in Instructional Technology requires a 
similar product during the student's final semester. Upon entry into an advanced degree program, students 
usually must enroll in one or more introductory cours es related to their profession. Seminars or modules 
devoted to the development of electronic files and use of public domain file transfer software (e.g. WSftp) will 
prepare students to master the basic technical skills needed for their portfolio. Use of a dedicated, and password 
protected, server is recommended for maintaining the database with all student portfolios. Access should be 
restricted to the student's personal directory and limited to faculty teaching in the program area. A screen shot of 
the database is displayed in Figure 1. Only faculty with administrator passwords can view the entire database. 
Students may provide prospective employers with the URL to their portfolio homepage and these can be viewed 
by any browser. 
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Figure 1 . First screen of database with all student portfolios. 



This ensures privacy for students and allows a systematic method for indexing student products that 
can be used for evidences needed for accreditation and program approval by state departments of education. 

The dedicated server eliminates the necessity for storing student work samples, semester after semester, on 
faculty hard drives and file cabinets. In addition, students may use the URL to their directory for personal job 
searches and promotions. The program areas described in this paper allows student access to their personal 
directory and files for a time period of up to three years after graduation. This provides an incentive for students 
to invest additional energy and creativity into the quality of their artifacts and design of the portfolio. 

Faculty from these programs report favorable responses from students. Students have adopted the use 
of portfolios as a method for reflectively reporting their personal reaction to course projects and assignments. 
Support for these conclusions come from (1) anecdotal comments from students and their advisors, (2) written 
responses to end of course surveys, and (3) reflective writings that justify student artifacts as representative of 
standards and competencies. Students have become increasingly more sophisticated in their knowledge and 
understanding in the portfolio as a tool for authentic assessment. Although, many professional programs 
include the “showcase” portfolio as part of the student’s culminating experience, a higher purpose for the use of 
portfolios is its use as a tool for constructed learning (Paulson & Paulson, 1994). The student must be able to 
articulate how his or her products reflect the criteria established by the standard. This facilitates transfer to 
actual working situations as the student enters his or her initial professional setting. 

As the instructor/assessor guides and provides council during the development of the contents of a 
student’s portfolio, there is opportunity for dialog and exchange of ideas. The student is able to see the 
modeling of professional behaviors, attitudes, and skills from a closer perspective than the usual interactions 
within the classroom setting. This is particularly important advantage for virtual classes and online degree 
programs. Indeed, the continual evaluation of assignments, and how these relate to professional standards, 
affirms the student’s professional goals, or in some cases, leads to consideration for a change in career paths. 

For the instructors and faculty of the program, there is opportunity for collaboration with colleagues to examine 
and evaluate program goals and objectives. Individual evaluation of program objectives and how these are 
reflected in course syllabi, activities, and assignments are a natural product of the process portfolio. 

There is growing evidence that portfolio assessment is a valid measure of skill and concept attainment, 
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and that there is reliability of measurement for predicting student achievement following graduation. However, 
research in this area of assessment is still limited. It is difficult to obtain data. There are misconceptions about 
the purpose and functions for portfolios, and authentic assessment requires a large investment of time. 
Additional time is needed for training of assessors and for counseling students. From this review and from my 
own observation of the portfolio process, the additional time needed is outweighed by benefits for student 
learning and for program improvements. The issues for validity and reliability should be considered before 
implementation. Faculty should clearly define any or all of the following: program objectives, national or 
professional standards and competencies, and performance indicators that represent the standards. These should 
be comrnu nicated to the student when entering the program. In addition, students should be advised and 
mentored with regard to quality of portfolio contents and how these reflect the specified standards and 
objectives. Finally, assessors should be trained in both consensus scoring and independent scoring procedures, 
and in determining a holistic evaluation of the final product. 
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